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Panel. From September 1998 to April 2000, the Panel met a total of five times 
to discuss its mandate (i.e., to evaluate educational programs and recommend 
to the Secretary of Education those programs that should be designated as 
exemplary or promising), create review criteria, establish a review process, 
and review programs . The report includes a look at the Panel process and the 
criteria by which submissions were judged. The following programs were 
recommended by the Panel and designated exemplary or promising by the 
Secretary of Education in 2000: (1) Maryland Virtual High School CoreModels 

Project; (2) Middle- school Mathematics through Applications Program (MMAP) ; 

(3) Modeling Instruction in High School Physics; (4) One Sky, Many Voices, a 
middle- school science program using real-time weather data; and (5) the WEB 
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reform in school systems throughout Vermont. Based on additional insights and 
knowledge gained from this undertaking, the report concludes with a section 
on further reflections by the Panel. (MES) 
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Preface 



This publication is the result of 20 months of work by the members of the Educational Technology Expert 
Panel. From September 1998 to April 2000, the Panel met a total of five times to discuss its mandate, create 
review criteria, establish a review process, and review programs. This report includes a look at the Panel 
process, the criteria by which submissions were judged, and the descriptions of programs recommended by 
the Panel and designated exemplary or promising by the Secretary of Education in 2000. Based on additional 
insights and knowledge gained from this major undertaking, the report concludes with a section on further 
reflections by the Panel. 
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Report on the Educational Technology 
Expert Panel 



Educational technology: A variety of electronic tools, media, and environments that can be used to 
enhance learning, foster creativity, stimulate communication, encourage collaboration, and engage in the 
continuous development and application of knowledge and skills. 

Introduction 

Educational technology has now evolved to a point where its power to transform schools is evident. This 
report, focused on the potential of technology to affect preK-12 student learning outcomes, is expected to 
advance the dynamic field of educational technology by helping 

o practitioners make better informed curriculum and budget decisions; 

o researchers design studies, assessments, and evaluations that are immediately relevant and meaningful 
to practitioners; and 

o policymakers shape and support initiatives that are well thought out and that include evaluation as an 
essential element. 

By 2001, public schools had invested almost $5.5 billion dollars in technology (Market Data Retrieval’s 2000- 
2001 Public School Technology Survey ), and 98 percent of public schools had access to the Internet 
(Internet Access in U.S. Public Schools and Classrooms : 1994-2000, NCES). Currendy, the critical ques- 
tions about educational technology for practitioners, researchers, policymakers as well as parents are: How 
can digital technology best be put to use in the service of precollege education? How can this technology be 
used to accelerate conventional learning as well as transform what students are learning and understanding? 
How can technology contribute to the acquisition of information as well as to the ability to assess reliability of 
findings and to synthesize incomplete but relevant data across several disciplines? How can technology enable 
students to demonstrate high standards of performance as well as to become contributing citizens of our 
increasingly global society and economy? 

The field of technology is in a state of constant change. Development has been uneven. Leaders in the field 
see pockets of excellence but also areas of genuine need (such as more universal individual classroom access 
to the Internet and home computers for more students), particularly on the part of underserved populations. 

Many in the education community are aware of the need for more guidance and discourse on how to maxi- 
mize any investment in technology. Many educators want to know how technology will make a difference to 
their students in the classroom. Trained to evaluate curricula and develop methods for teaching traditional 
disciplines, many educators feel unprepared to evaluate or use technology, yet they have an obligation to 
explain their technology expenditures to the tax-paying public by demonstrating how technology improves 
educational outcomes. The use of technology cuts across all academic areas, and it is changing rapidly, gain- 
ing greater capabilities seemingly every day. Most institutions — from businesses supported by resource-rich 
technology departments dedicated to mastering these powerful, expensive, and evolving electronic tools, to 
schools and school districts — are hard-pressed to keep current. Few general, informative, and stable guide- 



lines exist anywhere. Thus, concrete examples of successful technology use and thoughtful criteria can be an 
important source of practical expertise and information for others. 

In the field of education, the basic issues are: What is the appropriate role of the new technologies in educa- 
tion? How can the evolution of technology be directed efficiendy and effectively toward improving student 
learning for all? How can what is known be shared and developed further? The work of the Educational 
Technology Expert Panel is a beginning attempt to address some of these questions. Much more work 
remains to be done. 



The Expert Panel 

In 1994, Congress directed the Office of Educational Research and Improvement (OERI) to establish “panels 
of appropriate qualified experts and practitioners” to evaluate educational programs and recommend to the 
Secretary of Education those programs that should be designated as exemplary or promising. The purpose of 
these panels is to provide teachers, administrators, policymakers, and parents with reliable information about 
the quality and effectiveness of programs and materials so that they can make better-informed decisions in 
their efforts to improve the quality of student learning. Other expert panels have been convened for 
Mathematics and Science Education; Gender Equity; and Safe, Disciplined, and Drug-Free Schools. 

In 1998, OERI convened an Expert Panel on Educational Technology. The experts — drawn from universities, 
school-district administrations, preK-12 classrooms, research institutions, industry, government, philanthrop- 
ic organizations — encompassed a wide variety of backgrounds in using technology for education. 

After several months of in-depth discussions, and after an interactive process that solicited and used feedback 
from the field, the Expert Panel released its application and criteria for program submissions. The applica- 
tion and criteria are the result of a rare coming together of disparate, occasionally adversarial, points of view. 
Teachers insisted on the need for usable ideas from academe and practical support with technology in the 
classroom. Researchers insisted on the need for sound data, not anecdotes or hunches. Everyone agreed on 
the need for clarity — on the part of the Panel and on the part of programs for the purpose of dissemination. 

The Educational Technology Expert Panel recognized the importance of precise definitions of terms and 
objectives very early in the process. The Panel began by defining information technology as a tool with multi- 
ple potential applications for education, and agreed that learning to use this tool is necessary but is only one 
component of sound educational practice. In the past, technology discussions tended to focus on the infra- 
structure and hardware, such as how to wire schools or whether to put the computers in the classroom or a 
separate laboratory. Today the field has matured sufficiendy to focus on how to use the technology to improve 
teaching and learning. In fact, the objective of using educational technology should be to create an instruc- 
tional environment that fosters significant learning for preK-12 students. Technology affords new ways of 
teaching established disciplines that are likely to be more effective for more students; equally important, tech- 
nology can inspire teachers and students alike to move beyond conventional content and deepen the process 
of learning in various disciplines. Technology can be used to advance administrative and business functions 
in a school or district (e.g., to improve purchasing power, record-keeping, data-driven decision-making, 
communications with parents or between principal and faculty), which are important precursors to advancing 



student achievement. The Panel, however, decided that programs designated by the Department of Education 
should focus primarily on student learning. 

The Panel viewed significant learning as being multifaceted and active. It is a combination of skills, knowl- 
edge, and the ability to apply and communicate understanding as well as to learn further. 

What examples of technology programs working toward this kind of learning would be most useful as models 
to the field? The Expert Panel observed that with any endeavor involving technology, what is innovative today 
will be standard practice tomorrow. If the Panel had been assembled 2 years previous, the picture would 
have been very different. Then, the Internet was still a promising but exotic novelty; a technology plan meant 
dealing with equipment and was rarely linked to a plan for education. 

One lasting indicator of technological innovation, however, might be student learning that cannot be achieved 
without technology. Today, for example, schools are using the Internet to connect disparate communities that 
otherwise would have no opportunity to meet or interact. Another powerful use of technology is modeling, an 
aid to experimentation that seeks to understand the physical world. 

The Expert Panel further recognized that an exhaustive review of the many programs currendy active would be 
impossible. Compared to math and science, disciplines in which the content is fairly stable but approaches 
change over time — particularly in the early grades — educational technology is a fast-changing, dynamic tool 
that responds to emerging innovations and allows for many different approaches to many disciplines, often sev- 
eral at the same time. As the capacity of technology grows, so does the power of new programs. Thus, the 
Panel concluded that the most lasting, far-reaching guidance to the field would come from establishing useful, 
thoughtful criteria and dissemination strategies rather than from any particular programs it designated. 

The Expert Panel developed the six evaluation criteria and program indicators based on the four categories of 
criteria — (1) quality of program, (2) educational significance, (3) evidence of effectiveness, and (4) useful- 
ness to others — established by federal regulations for all panels. The Panel’s six evaluation criteria were 
carefully field-tested and resulted in the final review criteria presented on pages 7-10. 

The Panel determined that an exemplary program 

O addresses significant educational issues and identifies goals and a design supported by research; 

O improves preK-12 learning; 
o contributes to educational excellence for all; 

O promotes organizational change; 

o makes possible educational gains that cannot be achieved without the use of technology; and 
o serves as a model for other educational institutions because it is sustainable, adaptable, and scalable. 

The Panel also determined that a promising program also possesses these qualities, but may not yet have rig- 
orous and sufficient evidence for the program’s achievements. 
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The Expert Panel was not seeking programs that teach basic technology skills (how to use e-mail or the 
Internet) or that automate current classroom activities (“drill and practice”). Nor was the Panel interested in 
exciting programs developed by teachers in single classrooms because historically, successful scale-up of 
approaches idiosyncratic to an individual instructor have proved frustratingly elusive. Rather, the Panel 
focused on significant learning for preK-12 students and sought innovative, thoughtful programs that reflect 
systemic improvement within and across disciplinary boundaries with various student populations. 

From the Panel’s research review and familiarity with educational outcomes in the classroom, it was well 
aware that schools are complex organizations. Among the essential concomitants of increased student learn- 
ing are organizational change (e.g., scheduling of common planning time, new job titles such as technology 
coordinator, and prompt on-site technical support) and structural alignment (e.g., coordinating teacher pro- 
fessional development, curricula for inquiry-based learning, and assessment and curriculum standards) as 
well as an active commitment to equity. A truly powerful program benefits from contributions from different 
points of view and is successful with many kinds of learners, not just a narrow segment of students. 

In addition, the Panel had observed that the most effective programs are the most coherent; that is, they have 
a clear, purposeful linkage between actions and outcomes. The Panel attributed coherence to a clear articu- 
lation of the program’s objectives and conditions for success, combined with a constant, critical monitoring of 
evidence that supports or contravenes the program’s assumptions. The Panel further noted that useful evi- 
dence might consist of simple pre- and posttests, or the tracking of changes in student behavior (e.g., atten- 
dance and graduation rates) and is not limited to formal quantitative studies. 

The Panel believes that successful outcomes are the cumulative result of effort in multiple areas and has struc- 
tured its criteria to reflect that belief. 

The Panel’s definition of program goes well beyond that of a textbook or curriculum for a single discipline: 

Program: The implementation of a set of interrelated strategies and activities directed toward increased 
learning. A program includes assessment that demonstrates its impact on participants and on a wider 
organizational structure (i.e., a school or cluster of schools, a district, community organization, partner- 
ship, or other distributed system). Eligible programs depend on technology to achieve their goals. 

The Educational Technology Expert Panel believes that no one electronic device can fulfill its definition of pro- 
gram without a sustaining and complex structure of support. A program cannot be dissected and only certain 
components extracted and used — a program functions as an integral, interlocking whole. 



The Designated Programs 

The selection of exemplary and promising programs reflects the potential of educational technology for 
significant learning. Equally important, the programs serve to affirm and encourage the continued efforts 
of many others across the country who share a commitment to maximizing educational technology for the 
benefit of improved learning. 



Given that new projects are constantly being developed and that the Panel was dependent on the set of self- 
selected applications received, there is no assurance that the list of selected programs is fully representative of 
the variety of programs being implemented across the country. Rather, the list is illustrative; the programs 
exemplify excellence regarding certain features the Panel considered critical, including their effectiveness in 
exploiting the unique capabilities of technology and their contribution to professional development, equity, 
and assessment. The designated programs stand as models of the kind of thinking that leads to increased and 
sustained student learning. 

The process of identifying excellence is ongoing. The exemplary programs and many of the promising pro- 
grams can serve as models to all schools. The learning goals, methods, and design have universal application. 
Virtually any school with leadership and imagination can adapt these programs to their own populations, 
strengths, and weaknesses. The winners will be the students, teachers, administrators, and the community. 

How TO USE THE CRITERIA 

The Panel’s criteria reflect how experts in the field are currently thinking about technology and learning. The 
criteria can be used as guidelines when creating and improving programs that use educational technology. 
They provide a rigorous, analytical model for districts and schools undergoing planning and development. 
They are useful for policymakers deciding on how to benefit fully from connecting their buildings to the 
Internet, what objectives they want to achieve and with whom, what sorts of technology programs they want to 
create or adapt, what sorts of implementation issues must be considered, and what sort of assessment will be 
effective and persuasive. 

The criteria can also be used to explain to school boards, parents, and others why educators should bring 
technology into schools. They can also be used as an assessment guide to help schools scrutinize their cur- 
rent use of technology: How is technology currendy being used for student learning? Is equity addressed 
consciously and sufficiendy? Are systemic supports in place, and are more supports needed? 

The criteria reflect the experience of experts in the field, and, as such, can stimulate creativity and help 
avoid pitfalls. 

The Review Process 

The submission process was open, widely publicized, and designed to encourage applications from a variety 
of organizations. One hundred thirty-four programs were submitted for review. The review encompassed 
three stages. First, every submission was examined by a team drawn from 30 reviewers, each of whom had 
expertise in technology in addition to a content specialty such as math or an area of expertise such as school 
reform, research, or evaluation. This was the Quality Review Panel (QRP). Reviewer training included a 
series of online discussions with Expert Panel members over a period of 3 weeks, and then an onsite orienta- 
tion conducted by a team of 4 Panel members. These QRP teams determined eligibility and verified that the 
application was complete; scored the quality of the submission in accord with the Panel’s published criteria; 
and determined whether the submission merited being forwarded to the next group of experts — the Impact 



Review Panel (IRP). In the second stage, the IRP, consisting of national experts in evaluation design and analysis, 
assessed the persuasiveness of the evaluation data and the claims made by the program. Third, members from the 
Expert Panel reviewed all of the programs, along with the ratings and comments of the QRP and IRP review teams, 
to determine which programs to recommend to the Secretary of Education as exemplary or promising. 

The criteria used to judge the programs and the program descriptions for the two exemplary and five promising 
programs designated by the Secretary of Education follow. The descriptions are based on information provided by 
the developers. Contact information for each program is included. 



Next Steps 

The main goal of the Educational Technology Expert Panel was to develop and promulgate a shared vision, a com- 
mon framework, and specific definitions and criteria to raise the level of conversation and agreement within the 
field about educational technology in order to improve teaching and learning. Educational technology is still 
viewed by some as a matter of installing hardware and software, with insufficient attention being devoted to quality 
teacher training and evaluation. We hope that the Panel’s work will help policymakers, school administrators, edu- 
cators, and parents gain a more comprehensive view of the integral role that technology can play in curriculum 
development, organizational change, and school reform. 



Evaluation Criteria 

The diagram below presents the interrelatedness of the six criteria upon which programs were reviewed. 
Recognized programs have significant educational goals that result in complex learning supported by technol- 
ogy. These programs also promote organizational change as well as greater equity and educational excellence 
for all students. The programs demonstrate persuasively their effectiveness regarding these outcomes and are 
useful and adaptable in other school settings. 
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A. Quality of Program 



Criterion 1. The program addresses an important educational issue or issues and 
articulates its goals and design clearly. 

o The educational goals are significant, 
o The program design is thoughtful and supported by research, 
o The program description is clear and complete. 

Include the following items: 

a. Need or problem the program addresses and how it relates to teaching and learning 
in preK-12 schools. 

b. Program goals. 

c. Technology used and how it helps to achieve the program’s goals. 

d. Subject population (s): ethnic, racial, socioeconomic, and gender percentages. 

The size of any special populations served (e.g., ESL, AP biology students, 
students with disabilities). 

e. Content and learning goals. 

f. Program design (structure and components). 

g. Professional development provided as part of the program. 

h. Overall size and maintenance costs (funding and staff requirements, number of people 
in target population). 

i. Key learning activities for participants. 

j. Assessment(s) used to determine the program’s efficacy and achievements. 

k. Keys to the program’s success. 

l. A specific, concrete example that best captures the changes achieved by this program. 



B. Educational Significance 

The Expert Panel considers the following three areas — learning, equity, and organizational 
change — essential to fulfilling the promise of educational technology. A sound program must address all 
three, and all three must be shown to have impact on or linkage to preK-12 student learning. 

Criterion 2. The program develops complex learning and thinking skills for its target 

audience. 

If the target audience is other than preK-12 students, the applicant should 
articulate the program’s goals and their connection to student learning. 

If the target audience is preK-12 students, the indicators might include one 
or more of the following objectives: 
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o The program increases students’ in-depth understanding and competence in at least 
one content discipline. 

o The program develops the habits of lifelong learning (e.g., the ability to collaborate, 
direct one’s own learning, solve problems, communicate ideas clearly, and think 
flexibly and critically) . 

o The program helps students become proficient and critical consumers and producers 
of educational technology. 

o The program includes preparation for entering a technology-infused workplace. 



Criterion 3. The program contributes to educational excellence for all . 

o The program conveys high expectations for all learners, 
o The program responds to the diverse needs of varied populations of learners, 
o The program includes active outreach and partnerships to encourage broad 
participation. 

o The program increases the participation or achievement of underserved learners so 
that the gap between this group and other categories of students diminishes. 



Criterion 4. The program promotes coherent or ganizational chang e. 

o The program reflects a vision of educational reform consistent with disciplinary content 
standards, recommendations from national commissions, findings from educational 
research, and documented best practices, 
o Policies, funding, and practice are aligned to support sustainable change, 
o Through partnerships and professional development, the program builds human 
capacity to accomplish its goals (e.g., allocates time for teachers’ and administrators’ 
collaboration and planning). 

o The program increases the educational involvement of parents, professional groups, 
and communities. 



C. Evidence of Effectiveness 

Criterion 5- The program has rigorous, measurable evidence for its achievements for at 
least one criterion among Criteria 2, 3, and 4 (learning, equity, and 
organizational change). 

Valid evidence will meet generally accepted standards in the field and may include 



O one or more comparison groups; 



O a formal evaluation; 

O a quantified demonstration of positive change among participants as a result of the 
program (e.g., increased parental involvement in school governance; diminished gaps 
in achievement between groups; increased enrollment in rigorous mathematics courses 
or graduation rates among subject populations; changes in the base funding and 
requirements for professional development); and 
O an in-depth, qualitative analysis of change among participants as a result of the 

program (e.g., case studies, ethnographies, and principled analyses of observations and 
interviews). 



D. Usefulness to Others 

Criterion 6. The program is adaptable for use in multiple contexts. 

O The program’s technology requirements are easily available to potential users. 

o The program is cost-effective relative to its benefits. 

O After its initial implementation, the program is sustainable with existing resources 
(i.e., does not require extraordinary/unreasonable time, effort, or funding), and 
scalable (i.e., can naturally expand its scope to several teachers, multiple grade 
levels/subjects/sites, different disciplines). 

o The program is adaptable to a range of educational settings with learners similar to the 
intended population. 

O The program provides clear and detailed guidelines about the conditions required for 
its successful implementation. 



Note: The term “learners” can refer to preK-12 students, educators, or parents. 
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Program Description 




The Challenge 2000 Multimedia Project, created in 1995 and designed for students in grades K-12, 
establishes a learning community in which students use appropriate technology and acquire skills 
needed for the high-tech workplace — content knowledge, problem-solving acumen, communica- 
tion and collaboration skills, as well as self-assessment. On-site mentor-teachers support the 
design of classroom experiences in which students analyze and propose solutions for real-world 
problems by planning and producing interdisciplinary, multimedia projects. Students then present 
the final projects to the community at a multimedia fair. 

Support for teachers involves four components: (a) teaching and professional development 
intended to transform practice by helping teachers become effective designers and implementers 
of student-centered, technology-supported, project-based learning using multimedia; (b) comput- 
ers in the classrooms (rather than a separate computer lab) for all teachers and students; (c) 
classroom connections to the “information superhighway”; and (d) effective, engaging software 
and online learning resources as an integral part of school curriculum. 

Student multimedia projects might be a hypermedia stack, computer presentation, Web site, or 
video. Classes have studied such topics as the rain forest, water pollution, designing a research 
facility and living quarters for scientists in Antarctica, the physics of amusement park rides, and 
colonizing Mars. 

Quality and Educational Significance 

Learning 

Students develop in-depth understanding and competence in at least one content discipline. They 
also develop habits of lifelong learning (e.g., ability to collaborate, direct one’s own learning, solve 
problems, communicate clearly, and think flexibly and critically) . 



For example, at one school, 12th-grade physics students are paired and work to develop videos 
combining footage of amusement park rides and their associated forces and acceleration changes. 
The goal is to teach middle school and 9th-grade students the physics of the rides. The teacher 
observes that preparing a presentation for a younger audience really pushes the students to a 



Program Costs 

For cost information, please contact program designee. 



deeper level of understanding than would “spouting back calculus formulas.” Videos are judged on 
correct science content, the suitability of the material for the intended audience, and the effective 
use of multimedia to explore the planned content (incorporation of video, pictures, and data). 

The school holds an annual “Physics Day” when students collect data and images with video and 
digital cameras at the local amusement park. Students plan accordingly, working out what data they 
want to collect, how they will collect it, and how to share equipment. Sometimes they also share 
materials and calculations. To capture force in two of the three possible planes, each student pair is 
given two accelerometers and a graphing calculator with special software, which they program to 
collect the data they want before going on the rides. 

Back in the classroom, students upload their data and print out a graph of the data points. They 
can thus see patterns of acceleration associated with, for example, the spin of a car. One group of 
students chose to explain how roller coaster loops keep people in their seats, showing how the tear- 
shaped loops at the park worked and what would happen if the loops were circular. They calculat- 
ed that the force of a circular loop would make people six times heavier at the bottom than the 
force of a tear-shaped loop, so that circular loops were impractical. 

At the end of the spring semester, the videos are presented in a film festival and viewed on a full 
screen with high-quality amplification, accompanied by much celebration and cheering. 

Excellence for All 

Teachers report that where the implementation of project-based learning using multimedia has been 
greatest, there have been significant increases in teacher expectations of student work. Further, they 
report that low-achieving students benefit most from the project-based multimedia approach, fol- 
lowed by academically gifted and talented students. Other groups who particularly benefit are stu- 
dents with ADHD, English-language learners, and special education students. When adding teams of 
schools, the project has sought to increase participation by adding teachers from schools with the 
greatest number of disadvantaged students. 

Organizational Change 

Challenge 2000 advances organizational change by providing effective models of change and sup- 
porting the teacher learning community as they implement project-based learning in their class- 
rooms. It promotes the sharing of expertise by establishing partnerships focused around the imple- 



USEFULNESS TO OTHERS 

From 1995 to 1999, Challenge 
2000 increased its teacher cohort 
from 40 to 1 16. The project's 
technology requirements are 
affordable and realistic for most 
schools; the software is not propri- 
etary and can vary according to the 
resources of the school. Costs per 
unit are high during start-up but 
decrease as the program expands 
to a greater number of students 
and teachers. Participants include 
a wide range of ethnic, racial, and 
socioeconomic groups. 
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mentation of student-centered projects in classrooms. The partnerships are composed of a more experienced 
mentor teacher, who has previously implemented multimedia projects, and one to three teachers new to the 
project-based multimedia approach. Based on their ideas for projects, the partnerships have developed pro- 
posals to secure hardware, software, and network tools needed to implement their projects. During imple- 
mentation, teachers often discuss their projects within their school and have the opportunity to meet with 
teachers from other schools who have formed their own partnerships at program-wide meetings throughout 
the year. Their work culminates each year in a presentation of their students’ work at a Multimedia Fair, to 
which parents and community members are invited to witness the accomplishments of the teachers and their 
students. 

Evidence of Effectiveness 

Teacher reports of classroom practice and observations conducted in Multimedia Project classrooms and 
comparison classrooms show the impact of the program on teaching. Most teachers report implementing 
multiple projects with their students, and the more experience teachers have with Challenge 2000, the more 
likely they are to engage students in sustained inquiry — asking questions, investigating them, and making pre- 
dictions about phenomena. Students in these classrooms were more likely than those in comparison class- 
rooms to be engaged in long-term projects and to work collaboratively with their peers. They were also more 
likely to be engaged in cognitive activities of design — interpreting and analyzing information, deciding on the 
structure of a presentation, revising a presentation, and thinking about how their audience learns. 

In addition, the project conducted a performance assessment designed to measure students’ skill in construct- 
ing a presentation aimed at a particular audience, using content provided to them. Students from Multimedia 
Project classrooms outperformed comparison classrooms in all three areas scored by researchers and teach- 
ers who analyzed the student products: content, attention to audience, and design. 











Program Description 

Generation www.Y (GenY), started in 1996, focuses on today’s new generation of youth as partners, 
and often leaders, in bringing technology into the classroom. Instead of teaching technology skills 
to teachers, GenY trains students to partner with teachers to improve teaching and learning. Along 
with educators, students become agents of change, not mere recipients of change. The class has 
been implemented in grades 4-12, with the majority of participating students in grades 6-8. 

The core of the program is the GenY course, taught by a coordinating teacher and offered as an 18-week 
class in secondary schools or a 30-week class in elementary schools. GenY students are trained in com- 
puting and telecommunications skills for the purpose of helping rebuild the curricular units in their 
school so that teachers, administrators, and all students can make better use of modem technology. 

GenY students also receive training and coaching on communication and collaboration with teachers, 
project planning and management, standards-based curriculum development, and effective presentation 
of information to learners. Coordinating teachers participate in a one-day training workshop. 

One coordinating teacher and one class of students per semester are the kernel of the program in 
each school. Each GenY student in these classes forms a partnership with a teacher, known as the 
partner-teacher, and together the team designs and completes curriculum-building projects. Many 
teachers schoolwide participate as partners. The partner-teacher chooses a lesson plan or curricu- 
lum unit to upgrade or build from scratch, then teacher and student plan a project that enriches 
the unit or lesson plan, taking advantage of the available technology. The student focuses on the 
technical aspects of the project, meeting regularly with the partner-teacher to ensure that the proj- 
ect fits with the teacher’s curriculum plan and classroom needs. The partner-teacher uses this 
technology-enhanced lesson in classroom teaching, often continuing afterwards to develop or 
extend the project with continued technical assistance from GenY-trained students. The students 
provide Internet expertise (e.g., search out Web sites for information and help with electronic pres- 
entation of materials), while the teachers model skills such as communicating effectively, mentor- 
ing, solving problems, and project management. 

The GenY course is supported by an extensive curriculum guide, student workbook, CD-ROM, and 
videotape, all developed during the first 2 years of the program. Participants also have access to an 
online database of previous project descriptions, a planning database with facilities for aligning 
projects with state standards, and an online system staffed by content-area experts who provide 
feedback to students on their developing projects. 



Program Costs 



For cost information, please contact program designee. 



Quality and Educational Significance 

Learning 

GenY students receive training in network and Internet use, in a variety of computing skills, as well as 
in communication, collaboration, teamwork, project planning, project management, and project pres- 
entation. The training is designed to help students support teachers in rebuilding their lesson plans 
and curricular materials to enhance learning by using technology. 

The GenY experience involves a wide range of technical, academic, cognitive, and social skills. 

Students are expected to become proficient and critical consumers and producers of educational 
technology as well as to use these skills to help teachers use and produce educational technology. 

This requires understanding, then communicating and collaborating effectively with someone who may 
have a different point of view. When building collaborative projects, students must understand the 
needs of the partner-teacher, frame problems productively, seek out appropriate tools and informa- 
tion, then plan and manage their time and work in order to accomplish a discrete project within a 
specified timeline. Delivery of the projects entails good communication and presentation skills, using 
a style appropriate to the audience and the pedagogical purpose. The projects are typically unique 
and require students to think flexibly and creatively, direct their own learning, get help when needed, 
be responsive to feedback from the partner-teacher, and take responsibility for an academic and tech- 
nical product. One student, for example, developed a HyperStudio unit on prefixes, suffixes, and root j 
words that was easy for the class to understand and included a quiz that was fun to take. 

Benefits for the partner-teacher include an understanding of how to teach lessons enhanced by tech- 
nology and aimed at national and state learning goals and educational standards. In collaboration 
with the GenY student, the teacher develops a more collaborative relationship with students as 
coleamers and partners in the educational process, in which both student and teacher make impor- 
tant contributions. The GenY class is divided into 10 units that provide a suggested content and format 
for schools to implement the model. The format provides a model of project-based, authentic, stu- 
dent-centered, multidisciplinary teaching and learning enhanced by technology. 

Excellence for All 

The GenY model has been adopted in rural, suburban, and urban settings in several states and the Virgin 
Islands. As a result of outreach efforts during 1999-2000, the GenY program will be implemented in sev- 
eral inner city schools, western schools with a high Native American population, and in high-diversity 
schools in Hawaii. Because of a slight gender imbalance in GenY classes in general as of September 1, 

1999 ( 56-60 percent males vs. 40-44 percent females), one school has implemented special versions of 
the class aimed at appealing to girls. Program developers are studying this effort to determine how to 
improve the gender imbalance in other schools. All GenY students are held to the same standards: 
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Telephone: 888-941-4369, ext. Ill or 
360-528-2345, ext. 1 1 1 
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Usefulness to Others 

GenY began in the Olympia School 
District in the State of Washington. 
As of September 1 , 1 999, GenY 
has been adapted for use in 140 
schools in 1 1 states and the Virgin 
Islands. Several hundred more 
schools across the nation are 
beginning to use the model. After 
two years of intensive develop- 
ment efforts, the course is support- 
ed with an array of resources 
(some online) for coordinating 
teachers that can be adapted for 
different school settings serving all 
populations using virtually any 
hardware/software infrastructure. 
The course materials have been 
used successfully from upper ele- 
mentary through high school levels 
and include materials for adminis- 
trators, partner-teachers, coordi- 
nating teachers, and parents. The 
model is flexible, allowing classes 
to be conducted before school, 
after school, as a regular school 
elective during the day, or on 
weekends. 
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satisfactory completion of the curriculum units, including exercises, classroom activities, and assessments; delivery of a My 
articulated, planned project proposal during the middle of the semester; and completion and documentation of the project 
by the end of the semester. 

In addition to the core model, the program includes an opportunity for GenY graduates and preservice teachers to staff 
school-based or community computer labs. The labs provide increased computer and network access as well as 
coaching for underserved students, their families, and other community members. Students who use these labs often 
have no computer, no Internet access, or inadequate software at home, so the labs serve an important function in 
increasing the equitable distribution of access to technology. 

Organizational Change 

The premise of GenY is that students are a largely untapped, creative, enthusiastic source of partnership and leadership 
in an effort to use technology to support school reform. Thus, the program supports increased cooperation and collab- 
oration among students, teachers, administrators, family and community members, as well as business, educational and 
community organizations. The program requires coordinating teachers, partner-teachers, and students to think careful- 
ly about the alignment of curriculum units, lesson plans, and collaborative projects with state standards. In the state of 
Washington, GenY materials and online support are linked to the Washington Essential Academic Learning 
Requirements. Further, the online resources for project planning and reporting have been expanded to allow partici- 
pants in each state to work with their own system of learning goals and state standards. 

Evidence of Effectiveness 

GenY students, partner-teachers, and coordinating teachers reported that 90 percent of the students completed their proj- 
ects. These students generally feel proud of their projects, and their partner-teachers consider the projects to be of high 
quality (93 percent). The majority of the partner teachers (92 percent) reported using the projects in their regular classes 
and planning to update or extend the projects in future class activities (94 percent) . Almost half of the projects completed 
in spring 1999 were multimedia presentations and 25 percent were Web pages. The remainder involved teaching technical 
skills to teachers and other students, providing assistance in Web searching, and other activities. 

In spring 1999, 807 partner-teachers completed surveys before beginning the program, and 631 completed follow-up 
surveys at the end of the semester. About 90 percent of the responding teachers intend to use the new lessons repeat- 
edly or will continue to develop and expand them; this same percentage also reported that as a result of the GenY, they 
learned about technology and their students learned about technology and a content discipline. Nearly all (about 95 
percent) consider GenY a good method for providing support and assistance to teachers as they integrate technology 
into their classes, reported a desire to continue participating in the program, and said they would continue to rebuild 
lesson plans to benefit from using educational technology. Eighty-two percent of the teachers reported that the experi- 
ence would change the way that they teach in the future. 

When asked their opinion about using technology in education after participating in GenY, virtually everyone (98 percent) said 
they felt technology facilitates positive changes in classroom teaching and learning practices; 52 percent said they ranted to 
learn more about using new technologies; and 62 percent attributed these results to their participation in GenY. 

An external evaluation documents substantial learning gains on the part of participating students. Reviewers were 
impressed by the creativity of the project, creating a role reversal in which students help support the school's technolo- 
gy infrastructure and partner with teachers in curriculum development. The latter is crucial to the success of the proj- 
ect and to fostering learning gains for all students in participating districts. While a few projects have taken similar 
approaches, this particular implementation is better conceived, more thoroughly implemented, and more careMy doc- 
umented than other comparable programs. 
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The Maryland Virtoal High School 
(MVHS) CoreModels Project 
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The Maryland Virtual High School (MVHS) CoreModels Project has two primary goals: (1) to use 
computer modeling to help all high school students achieve state and national science standards, 
and (2) by developing and refining a process of network-based peer leadership and collaboration, 
to give teachers in their classrooms the support they need to integrate modeling activities into their 
instruction. Started in 1997, CoreModels is based on, and is intended to institutionalize, the gains 
achieved by the Maryland Virtual High School program that connected rural and underserved 
Maryland schools to a magnet high school via the Internet. Through this connection, MVHS sup- 
ported teachers in developing and implementing computational science projects with their stu- 
dents. Crucial components of the MVHS program were: (1) its focus on using computational 
modeling to teach complex science content, and (2) its mentoring and support for remote teach- 
ers and students provided by magnet teachers and their students. The mentors won national 
recognition for their computational science projects and gained experience through a student-run 
laboratory at Montgomery Blair High School Magnet Program. 




Program Costs 



A model of distributed support that creates the conditions for sustainability of the changes and 
gains achieved by schools, the program is active in 6 to 15 schools in each of 3 CoreModels 
regional centers in Maryland. 

Students construct their own models, utilizing the software STELLA, in areas of science such as 
wildlife populations, the carbon cycle, hurricane prediction, projectile motion, chemical reactions, 
and rock formation cycles. Students hypothesize about the results of their model based on the 
parameters used in the graphical model definition. They compare their results with other predic- 
tions, such as those from scientists or developed by other students, or with actual results as meas- 
ured in practice, thus gaining an understanding of important recurring scientific concepts involving 
equilibrium processes, feedback, and multiple causal relationships, among others. 

This process of learning brings students closer to science as it is practiced in the world outside of 
school. As an example of the conceptual core of the use of modeling, students not only learn 
about the laws of conservation of mass and conservation of energy, but they learn to apply them in 
many different situations; to identify control and independent variables; to collect data and analyze 
the results; and to write clear step-by-step instructions for investigating their hypotheses. In the 



For cost information, please contact program designee 
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process, students apply higher-level mathematics skills, such as ratio and proportion and setting up 
equations. For example, in the glucose regulation model students simulate the glucose-insulin feed- 
back process that the body uses in attempting to maintain homeostasis (equilibrium) and predict 
the effect on the body of eating a candy bar. They practice graph analysis by determining how the 
model output differs from their prediction and describe what the graphs mean in terms of the body’s 
response. They also use the graphs to describe how the body’s blood glucose and insulin concen- 
trations differ when pasta is eaten instead of a candy bar. 

The MVHS CoreModels project works through regional centers to integrate curriculum reform, staff 
development, and teacher professionalism in a sustainable manner. Technology enhances and facili- 
tates this initiative by allowing educators and students to become better at teaching and learning. 
Teachers are recruited and trained to 

O implement computer modeling to help students understand and learn more complex 
scientific concepts central to biology and physics; 

O help students learn in a collaborative, problem-solving, interactive manner that brings 
them closer to the world of science; and 

O implement innovative pedagogical practices by being provided with scientific and technology 
support. 

Ideally, as more teachers are recruited and trained, the pool of trainers for new teachers becomes 
larger, and more and more teachers become trained in the use of CoreModels. The project had 
been in existence for 2 full years as of September 1, 1999- 

Quality and Educational Significance 

Learning 

MVHS CoreModels increases students’ in-depth understanding and competence in their current sci- 
ence subject area by focusing on an analysis of change over time and on the transfer of understand- 
ing from one activity to the next. One of the hallmarks of the project is the alignment of CoreModels 
activities with Maryland Science Core Learning Goals (CLGs). CoreModels materials also focus on 
the American Association for the Advancement of Science (AAAS) Benchmark common themes 
(similar to National Science Education Standards themes), which emphasize connections between 
seemingly disparate science content. 

In using and creating computer models, student attention can be focused on similar structures and 
behavior across areas of study. For example, a predator-prey interaction model and a physical 



For Further Information, Contact: 



Mary Ellen Verona or Susan Ragan 
Maryland Virtual High School 
CoreModels Project 

Montgomery County Public Schools 

51 East University 

Silver Spring, Maryland 20901 

O 




Telephone: 301-649-2880 
Fax: 301-649-8245 



E-mail: mverona@mvhs.mbhs.edu or 
/! sragan@mvhs.mbhs.edu 
Home Page: http*y/mvhs.mbhs.edu 



Usefulness to Others 

The very structure of MVHS 
CoreModels lends credence to a claim 
of usefulness to others. The project 
includes regional centers, 1 5 local 
educational authorities, and 31 
schools. Growth and dissemination 
have been extremely successful in 
regions with very different demo- 
graphic and political characteristics, 
and the program has been adapted 
differently on the school, district, and 
regional levels. 

The program's technology require- 
ments are relatively simple, under- 
standable, and affordable. Schools 
can purchase the site license for STEL- 
LA software for $500. Unlike subject- 
specific simulation software, STELLA is 
technology problem-solving software 
(National Educational Technology 
Standards). In addition to computers 
running STELLA, Internet access 
should be available for teachers to col- 
laborate with peers and to access 
models and materials. In adapting the 
MVHS CoreModels program, leaders 
could access materials available 
through the MVHS site, as well as 
other system dynamic modeling 
archives such as the Creative Learning 
Exchange. 
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spring model share the same mathematical oscillatory structure. Disruption and resumption of equilibrium 
can be found in both biological and Earth systems. Assimilating an understanding of such core structure and 
behavior concepts leads to acquisition of the “schemas” of science content, which have been shown to distin- 
guish experts from novices (P. Chandler &J. Sweller [1991], “Cognitive Load Theory and the Format of 
Instruction,” Cognition and Instruction 8, 293-332). 

Excellence for All 

Modeling activities have been used with mainstreamed special education students as well as with advanced 
placement students. MVHS CoreModels developed Earth science and environmental science activities because 
these classes often target students considered less successful in previous science courses. In addition, center 
directors, who observe all teachers within their region, help teachers develop higher expectations for students 
in nonacademic classes by suggesting visits to teachers already successfully using activities with similar groups 
of students. 

One reason exemplary science activities are not available to all students is that only a fraction of eligible teach- 
ers take the initiative to investigate and apply to summer training programs. Facilitating organizational change 
through the CoreModels Centers is needed to provide training for additional teachers and learning opportuni- 
ties for all their students. The group of teachers thus targeted typically teach the non-AP and nonhonors classes 
so that affecting the practice of these teachers is one step toward contributing to educational excellence for all. 

Organizational Change 

One vital component of the MVHS CoreModels project is the role of the teacher in developing activities and 
evaluating their effectiveness in the classroom. Extending the initial success of MVHS in developing a remote, 
network-based collaborative learning environment, the CoreModels program supports teachers’ ability to 
change and integrate curriculum and technology through the use of peer mentors. 

Capacity building is another important aspect of MVHS CoreModels. The project is deeply committed to the 
development of leadership among participating teachers; its predecessor, MVHS, was a teacher-developed, 
teacher-led program. Although the CoreModel program’s institutionalization goals call for a leadership team 
(the project director, three regional center directors, and eight supporting teachers), the participating teachers 
collaborate with leaders in presenting outreach workshops and in developing materials. This expanding lead- 
ership has allowed the project to respond to opportunities to create districtwide and school-based training 
opportunities as well as to reach out to all teachers, meeting them where they are. Teachers are immersed in a 
profound form of professional development when they work together to adapt materials for the specific needs 
of a particular class; when they prepare a presentation for a group of colleagues; or when they think through 
how modeling can be used to facilitate and assess student learning in a new content or skill area. 

Evidence of Effectiveness 

Preliminary results show that learning appeared to have been enhanced by the use of the CoreModels project. 
Students in one science class using the CoreModels project improved their scores 46 percent on the Force 
Concept Inventory test, while students in another school, regarded as the best in the county but not using the 
program, improved by only 27 percent. In addition, a school with all its biology teachers involved in the 
CoreModels project led its district in the initial results of the state biology exam. 
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Middle-school Mathematics through 
Applications Program (MMAP) 

Program. Description 




The Middle-school Mathematics through Applications Program (MMAP) is a comprehensive set 
of middle school materials based on projects. The projects, which utilize software tools, 
engage students with real-world problem solving and require them to learn mathematics and to 
apply what they have learned. MMAP, started in 1992, addresses the needs of students in 
grades 6 through 8 with a series of units that emphasize proportional reasoning, algebraic 
expressions, and functions. It also covers statistics, probability, measurement, and geometry. 
The materials are designed to increase students’ conceptual understanding of mathematics, 
their competence with standard symbolic notations for mathematical concepts, and their ability 
to communicate their mathematical ideas to others. They also learn mathematical practices 
such as making conjectures, coming up with counter examples, and writing proofs. 

For example, with the design unit Antarctica and the software ArchiTech, sixth- and seventh- 
grade students play the roles of architects who design an ideal living space for themselves or a 
client; seventh- and eighth-graders design a scientific research station in Antarctica. Thus, stu- 
dents learn about scale and standard proportional notation; learn to represent the real world 
symbolically; research Antarctica and the needs of their clients; consider the building and heat- 
ing costs of their design; and present their floor plans, budget, and a written report to their 
class. These same skills are approached from a different perspective with the design unit 
Guppies and the software Habitech, for which students play the roles of population biologists 
who research, analyze, model, and make predictions about guppy behavior. They research 
behavior, reproductive habits, and life cycles of guppies; organize their data; model guppy 
breeding and feeding behavior; design a sustainable tank environment for guppies; and share 
their findings and suggestions with each other. 



Quality and Educational Significance 

Learning 
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Central to MMAP is the way in which the program engages students by placing them in software 
environments to solve open-ended design problems requiring the use of mathematics concepts. 
For example, Coding Toolbox allows students to use shift cipher, function, matrix, analysis, and 



Program Costs 

For cost information, please contact program designee. 
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compression tools to develop and test privacy and efficiency codes. The software complements, rather 
than dominates, the classroom activities. As students strive to improve their designs, they find them- 
selves using sophisticated combinations of math tools, representations, and techniques. 

By applying concepts to find solutions to relevant problems, MMAP students appreciate the power of 
mathematics. Their responses can be mathematically complex as well as tied to real concerns. Real- 
life contexts for learning and work-world technology tools help prepare students for the workplace. 

MMAP’s use of design processes, ongoing embedded performance assessments, and technology-based 
tools help students make explicit links between informal and formal mathematics. In the course of 
the Antarctica unit, for example, students are asked to compute the average yearly temperature in 
coastal Antarctica (computational skills, developing equations); furnish their floor plan design using 
resizable furniture icons in the ArchiTech software (scale and proportion); as well as graph and com- 
pare the relationship of building costs, heating costs, and insulation costs (patterns and functions). 
Other activities include building biological models and building digital maps with data layers. 

Excellence for All 

The overall design of MMAP is intended to engage more students successfully with mathematics, par- 
ticularly students traditionally underserved in school math or those who find themselves alienated 
from it. The software and topics were designed to be equally unfamiliar but engaging to most middle 
schoolers (i.e., the simulation context of Antarctica, the professions of population biologist or archi- 
tect). In addition, the technology is tool-oriented and relies on multiple representations; prior com- 
puter or math savvy is not necessary to achieve results. 



Usefulness to Others 

MMAP has been successfully inte- 
grated in many settings and is cur- 
rently used in racially, economically, 
and linguistically diverse communi- 
ties in California, Alaska, New 
York, Florida, Michigan, Oregon, 
and Washington. It also serves 
students with learning disabilities. 

The program is most successfully 
implemented in schools where stu- 
dents have constant access to a set 
of mathematical tools, including 
calculators and measuring devices, 
and where each MMAP class has 
access to at least four to six com- 
puters. 



The design process itself encourages the participation of many students as well as multiple solutions to 
a problem. Design begins with brainstorming, welcoming the different skills of different students and 
encouraging students to bring their personal experiences, skills, and identities to the task. For exam- 
ple, while working with ArchiTech on house design, one student compared the group’s proposed 
design to the size of his own house and realized it was huge; the group revised its floor plan. Students 
work with others to learn what they need to know, rather than being singled out for not knowing 
something. 

Organizational Change 

Among MMAP's greatest accomplishments is that teachers — both those who are codevelopers as well 
as those who use the materials at a distance but participate in the ongoing online discussions and sup- 
port activities — have rejuvenated their careers. Teachers report varied experiences that have helped 
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them change and make changes. For some, it is seeing extraordinary things happen for students as they partici- 
pate in new ways. Some are empowered when they start to use technology and find that it enables them and their 
students to improve. Teaching to standards and teaching with technology open teachers to new possibilities for 
student learning. 

Evidence of Effectiveness 

MMAP encourages flexibility in the ongoing assessment process, allowing teachers to gear their assessments to 
meet the individual needs of their students. The assessment process stresses the importance of a variety of means 
of communicating student understanding. Options include pre- and posttests, journals, log books, oral presenta- 
tions, writing, peer reviews, teacher-student conferences, group meetings, and more. Progress reports can be 
given almost daily, affording the student immediate feedback. 

A 1994 external evaluation surveyed 24 teachers and 42 students on program perception. Students, especially 
from traditionally underserved populations, indicated excitement and high motivation. Teachers also reported 
notable positive effects of MMAP on students’ various skills in both cognitive and affective domains and on stu- 
dents’ understanding of some mathematical topics. Results indicated a positive perception towards the program 
that was similar across gender and special education status. A 1995 teacher survey of 10 teachers provides some 
evidence that teachers have positive perceptions of the program and that they believe the program is beneficial to 
students. 

An internal evaluation utilized classroom observation and videotape analyses, reviewed school test data from exist- 
ing accountability assessments, and reviewed report card data to demonstrate student concept development and 
mathematical power (single group design). In 1997, program students in one classroom showed higher scores 
on algebra readiness tests in comparison to the same teacher’s students in previous years with less usage of MMAP, 
and a 100 percent passing rate at mid-year in eighth-grade algebra, in comparison to the previous school average 
of 50 percent passing at mid-year. Anecdotal evidence also supports the claim of students’ growth in mathematical 
understanding and improvement in student attitudes towards learning mathematics. 
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PROMISING PROGRAM ^ 

Modeling Instruction in High 
School Physics 

Program Description 




Modeling Instruction in High School Physics, started in 1990, uses computers to teach models and 
modeling, central components of modem science. These components are focal points to develop 
the content and pedagogical knowledge of physics teachers, who then serve as local experts on the 
use of technology in teaching and learning science. Science, and physics in particular, is a content 
area for which students need to learn how to use computers as a scientific tool for observation, 
data acquisition, analysis, and problem solving. Teachers are trained to support technology-based 
learning in up to 8 weeks of intensive Modeling Workshops conducted over two summers, and 
with ongoing year-round electronic network support. Teachers are thus engaged in a complete 
revamping of high school physics to incorporate both technology and the insights of educational 
research in full accord with the National Science Education Standards. The training provides them 
with a robust new teaching methodology that greatly increases students’ understanding of basic 
physics. 

In the Modeling Workshops, participants are introduced to modeling as a systematic approach to 
the design of curriculum and instruction. Teachers identify a small number of models around 
which to base their physics course and learn strategies to help students develop those models. 
They collaborate on the redesign of the high school physics course to enhance learning and 
employ technology to achieve their goals. They learn how to use computers as an integral part of 
their teaching practice. They implement a student-centered instructional strategy which engages 
students in active scientific inquiry, discourse, and evaluation of evidence. Further, they examine 
the implications of educational research for physics teaching. They do all this while immersed in 
studying the content of the entire year, which also provides extensive remediation for underpre- 
pared teachers. 
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The participating teachers then become leaders at their schools, modeling the best use of technol- 
ogy in the science classroom, training and assisting other teachers in the scientific use of technolo- 
gy in instruction, and advising schools on cost-effective infusion of technology into all science 
courses. These leaders must remain practicing science teachers, not simply technology specialists, 
because effective instruction in the use of computers as scientific tools requires special compe- 
tence in pedagogy and science as well as in technology. 



Program Costs 

For cost information, please contact program designee. 
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Note: Modeling Instruction in High School Physics was designated exemplary by the 
U.S. Department of Education's Mathematics and Science Education Expert Panel, 2000. 



Quality and Educational Significance 



Usefulness to Others 



Learning 

Teachers learn to 

O organize course content around a small set of basic models as the content core of physics; 

O engage students collaboratively in making and using models to describe, explain, predict, 
design, and control physical phenomena; 

O involve students in using computers as scientific tools for collecting, organizing, analyzing, 
visualizing, and modeling real data; 

O provide students with basic conceptual tools for modeling physical objects and processes, 
especially mathematical, graphical, and diagrammatic representations; 

©show how scientific knowledge is validated by engaging students in evaluating scientific 
models through comparison with empirical data; 

O assess student understanding in more meaningful ways and experiment with more authentic 
means of assessment; 

O improve continuously and update instruction with new software, curriculum materials and 
insights from educational research; and 

Owork collaboratively in action research teams to mutually improve their teaching practice. 



For example, in one experiment, students are asked to develop principles for the motion of a pendu- 
lum. With the teacher as recorder, students brainstorm about properties of the pendulum which 
might affect its period. After compiling the list, teacher and students decide which properties should 
be investigated. In this example, they determine to investigate how changes in mass of bob, length of 
string, and amplitude of motion affect the period. Students then work in teams and determine their 
own procedure for collecting data. After collecting data, they plot variables appropriately and then 
elicit the equations of motion and relationships among the variables. Then, using a technique called 
“whiteboarding,” groups present the results of their experiments to the class. At the end of this 
process, the class can agree on an appropriate model to describe the behavior of the pendulum. 
They do this without being given the answer by a text or a teacher. 

Students learn to understand scientific claims and to make sense of their experiments themselves. 
They must articulate coherent opinions of their own, defending their findings in a variety of formats, 
using concise English sentences, graphs and/or diagrams, and through algebraic expressions of the 
relationship. Students are forced to reflect on why they choose an answer and how they evaluate 
evidence. 



For Further Information, Contact: 



Jane C. Jackson 

Modeling Instruction in High School 
Physics 

i . , ; 

Box 871504 

Department of Physics and Astronomy 
Arizona State University 
O je, Arizona 85287-1504 

ERJC ' 



Telephone: 480-965-8438 
Fax: 480-965-7331 

E-mail: jane.jackson@asu.edu 
Home Page: http://modeling.asu.edu 




The Modeling Instruction Project has 
demonstrated its adaptability for use 
in multiple contexts. It began with 
pilot workshops in Arizona, where it 
has flourished for the past decade in 
two dozen urban, suburban, and 
rural high schools. It is now used in 
44 states in diverse schools, includ- 
ing impoverished urban public and 
exclusive private preparatory high 
schools. Its practitioners include 
experienced high school physics 
teachers with exceptional qualifica- 
tions and motivation as well as 
underprepared teachers who were 
drafted into teaching physics. More 
than half of the physics teachers in 
Arizona have learned Modeling 
Instruction, and the project's 
approach is making inroads in sever- 
al other states. 
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Excellence for All 

The project developed three new evaluation instruments and found that the program increases the achievement 
of underserved learners. For example, 36 Phase 1 teachers (teachers who had participated in one summer 
workshop) were found to fall into one of two groups: those implementing all aspects of Modeling Instruction 
consistently (17 teachers), and those implementing some aspects of modeling consistently (19 teachers). In 
disadvantaged/lower income schools, the mean FCI gains of the 93 students of the 2 teachers who implemented 
the Modeling Method consistently were 25 percent higher than for the 335 students of the 6 teachers who were 
implementing less modeling. Students of both groups did better than those in traditional courses; the FCI gains 
of the 93 students were double that of students under traditional instruction, and the FCI gains of the 335 stu- 
dents were 50 percent higher than those under traditional instruction. 

Many teachers, including inner-city teachers, report that their enrollments have increased since they started 
using the Modeling Method. 

Organizational Change 

Graduates of Modeling Workshops are in great demand in their schools to assist in incorporating technology in 
other science and math courses. In consequence of its well-documented success, the project has stimulated 
formation of the Arizona Science and Technology Education Partnership (AzSTEP), which is institutionalizing 
the reforms and methods of the Modeling Workshops in a statewide program for professional development of 
in-service physics teachers as local leaders of science teaching reform. AzSTEP is an exemplar of a university- 
high school partnership to drive sustained reform of science teaching with technology. 



Evidence of Effectiveness 

An internal evaluation of data from 1995-1999 utilized a pre/posttest comparison design with matched stu- 
dents on the Force Concept Inventory (FCI) test, a test used to determine the effectiveness of mechanics cours- 
es in teaching students to reliably discriminate between the applicability of scientific concepts and naive alter- 
natives in common physical situations. The nationwide sample contained approximately 10,000 high school 
physics students involved in the program. The comparison groups consisted of approximately 8,000 high 
school physics students of the same teachers in the year before the teachers began the Modeling Workshop 
series and 700 high school students of teachers in traditional courses. Results indicate that Modeling 
Instruction students demonstrate greater gains on the FCI than comparison students. In the nationwide sample 
of students, the average FCI pretest score was approximately 26 percent, slighdy above the random guessing 
level of 20 percent. Students of traditional high school instruction (lecture, demonstration, and standard labo- 
ratory activities) achieved an average posttest score of about 42 percent. In contrast, Modeling Instruction stu- 
dents recorded posttest FCI scores of approximately 53 percent when the teacher was in her first year of the 
program and an average of 69 percent when the teacher was an expert teacher with 2 years of modeling expe- 
rience. 

An external evaluation utilizing classroom observation of teacher training workshops determined that there was 
“overwhelming and consistent support” for this teaching approach. In interviews, participants reported that the 
workshops were well run and that the facilitators were extremely knowledgeable about how best to teach 
physics. They also commented that the group itself was an excellent resource as they could all help each other. 
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Program Description 




One Sky, Many Voices (OSMV) is a research-focused learning program based on the student's appli- 
cation of knowledge to solve science problems using real-time weather data. Visualization software 
allows students to track real-life events such as hurricanes, blizzards, floods, and tornadoes. The 
program, started in 1992, covers 4 or 8 weeks of coordinated study in middle school science, 
teacher support in the form of local study groups and focused networked discussions by experienced 
teachers, daily scientists’ updates, and a suite of state-of-the art technological tools including current, 
customizable weather imagery and message board systems. OSMV has been implemented, with vary- 
ing levels of support, with a large number of students in impoverished urban schools as well as sub- 
urban and isolated rural schools. Science- and data-based interactions among students from these 
diverse populations is a strength of the program. 

OSMV promotes important attributes associated with significant learning experiences. Middle school 
students use an Internet browser for enhancing their investigations of current weather phenomena. 
Guided by their teachers and linked with experts in the atmospheric and environmental sciences, stu- 
dents view real-time images and converse with their peers on Web-based message boards as an inte- 
gral part of the their studies. Thus, students might experience tracking and predicting current hurri- 
canes, collaboratively studying and discussing current weather fronts in their region, or developing 
content-rich explanations of weather phenomena in their area to be shared with students who do not 
live with the same extreme weather patterns. 

Studying real-time weather as part of One Sky, Many Voices has afforded students the opportunity to 
become involved in real-life occurrences. For example, after tracking Hurricane Mitch, many stu- 
dents helped organize relief efforts for victims of the weather phenomenon that they had come to 
know so intimately as part of their studies. 

One Sky, Many Voices engages and empowers students as scientists and provides support for teachers 
using the associated technology and pedagogy. Students take an active role in the teaching and learn- 
ing experience. The exchange of roles between students and teachers in this program reflects a 
sound, progressive school reform effort. 




Program Costs 



For cost information, please contact program designee. 
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Quality and Educational Significance 



Usefulness to Others 



Learning 

Teachers learn to 

O provide students with basic conceptual tools for understanding the use of modeling 
processes, especially in visual, mathematical, and graphical, representations; 

O assess student understanding in more meaningful ways and experiment with more authentic 
means of assessment; 

O improve continuously and update instruction with new software, curriculum materials, and 
insights from educational research; and 

O work collaboratively in action research teams to mutually improve their teaching practice. 
Students learn to 

O engage collaboratively in making and using models to describe, explain, and predict physical 
phenomena; 

O understand how scientific knowledge is validated by evaluating scientific models through 
comparison with empirical data; and 

Ouse computers as scientific tools for collecting, organizing, analyzing, visualizing, and 
modeling real data. 

Students increase the abilities necessary to conduct and understand scientific inquiry as well as to 
make sense of their predictions themselves. They hone their communication skills while conducting 
online interactive discussions with content mentors and peers distributed nationwide on their focus 
topics. Peer explanations and predictions are critiqued and discussed in a group forum. After partic- 
ipation in OSMV, students display knowledge of significantly more weather terms, more scientifically 
valid claims, and more sophisticated measures of scientific thinking. 



Between 1992 and 1999, OSMV 
grew from 6 teachers and 125 stu- 
dents to over 240 teachers and 
1 1 ,000 students. Its designers start- 
ed small in order to research what 
models of teacher support, student- 
learning support, and emerging 
technologies really worked for a 
wide range of classrooms and audi- 
ences. They believe that the key to 
the program's success is its respon- 
siveness to the needs of teachers 
and students. Although OSMV is 
concentrating its efforts on the par- 
ticular challenge of urban class- 
rooms, the program, with its 
focused use of the Internet and cur- 
riculum combined with full support 
for teachers, fulfills a need 
expressed by many teachers every- 
where. 



This integrated program was specifically designed to help learners of all backgrounds, whether students, 
teachers, researchers, or scientists, to use each other as resources and to collaboratively study interdiscipli- 
nary science and its impact on humanity. This is in contrast to the use of curriculum materials in isolation. 
OSMV is also intended to help learners of all backgrounds better understand the tremendous power of the 
Internet, when utilized well, to afford profoundly rich and meaningful learning experiences. 



For Further Information, Contact: 



Nancy Songer % 

One Sky, Many Voices 



Telephone: 734-647-7369 



Fax: 



734-763-1368 



The University of Michigan 
1323 SEB/610 East University 
Ann Arbor, Michigan 48109-1259 



E-mail: songer@umich.edu 

Home Page: http://wwvv.onesky.umich.edu 

\ 
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Excellence for All 

OSMV achieves its goals by My supporting teachers in bringing to their community and their colleagues the 
kind of different learning that Internet-rich programs can offer to schools. OSMV targets schools with low con- 
nectivity and that face similar challenges as a way to best meet the needs of all schools. The program currently 
focuses its local activities on schools in Detroit, many of which have very low socioeconomic status, high crime 
rates, and challenging teaching environments. 

Organizational Change 

Locally, teachers form study groups before, during, and after curricular implementation as a means of scaffold- 
ing the learning of new teachers. Because many of the teachers are distributed nationwide, OSMV also organ- 
ized similar scaffolded study groups online and compensates returning OSMV teachers to act as discussion 
moderators, to seed discussions, and to respond to the questions and issues of other teachers. After several 
years of OSMV implementation, a strong cohort of returning teachers serves as an extremely valuable resource 
for new teachers and sees their professional and in-school roles change. Content support is facilitated through 
a “message of the day” posted daily by OSMV scientists to assist teachers in guiding student learning to an inter- 
esting current weather event of the day. This allows teachers to be content-sawy about scientific phenomena 
not normally covered in traditional textbooks and makes the content more interesting because of its real-life 
ramifications. 



Evidence of Effectiveness 

OSMV makes a measurable difference in learning. In 1997, one study found that Detroit seventh-graders in the 
program (95 percent African-American) not only outperformed eighth-grade African-American students nation- 
wide on content addressed in the curriculum but outperformed all eighth-graders nationwide on these items. 
These same Detroit students performed significantly lower statistically than all eighth-graders, including 
African-American eighth-graders, on items not addressed in the program. 

The research results also demonstrate that both students and teachers become significantly more computer-lit- 
erate after work with our programs, and that their attitudes about both science and technology are improved 
after participation. 

OSMV programs have been implemented with urban, rural, suburban, and homeschooled populations. In one 
important measure of effectiveness, girls’ confidence in science increased significantly to match that of boys at 
the end of the program. In addition, girls indicated an increase in expectations of achievement at the end of 
the program. 

Teacher comments speak eloquendy to the program’s effectiveness: “Very valuable. My students even gave up 
their recess to read messages.” “I can’t say enough about Hurricanes. . . The people who put it together really 
understood what teachers need, and need at their fingertips. The kids were on fire with how much they were 
learning and they hated to quit.” 




The WEB Project 

Program Description 

The WEB Project created a consortium of community organizations, small businesses, and educa- 
tional institutions, which collaboratively learned how to employ new technologies to effect systemic 
reform in school systems throughout Vermont. The Project, based in Putney, Vermont, is designed 
for students in grades 2-12. It utilizes multimedia production and telecommunications as: (a) an 
educational environment for student inquiry and expression; (b) a medium for presenting and 
assessing student work; and (c) a virtual faculty room for professional discussions about work. 

Begun in 1995, the WEB Project has developed systems for the dynamic use of multimedia and 
telecommunications to improve student learning in the Vermont Vital Results — communication, 
problem solving, civic responsibility, and personal development — as they apply to arts, language, lit- 
erature, history, and social studies. Partnership initiatives include sharing music compositions, art 
projects, literature, and historical research online within the context of specified learning goals. 
Each initiative specifies standards that students will reach as a result of participation. 







Quality and Educational Significance 

Learning 





The WEB Project uses current multimedia technology as a tool to provide opportunities for students 
to gather rich source materials; solve problems focused on content; critique their own work as a nat- 
ural part of the production process; determine trade-ofiEs between time, quality, and technological 
limitations; create new knowledge; and express themselves artistically. Several studies cited by the 
Web Project indicate that the multimedia and music composition activities lead to complex learning 
and provide evidence that student learning is improving in important and complex areas. 

Multimedia production requires teamwork. Collaborative problem solving and design adds a 
dynamic, real-world connection to student experiences. 

Through the WEB Project process, students learn how to realize their intentions more fully by learn- 
ing, for example, how to differentiate foreground from background more clearly; how to use shading 
to give a drawing more dimension; and how to focus the viewer’s attention on the important elements 
of a drawing or sculpture. In the virtual faculty room, for example, teachers discussed a disturbing 
drawing (“Cop Killer”) posted by a student. They covered the issue of censorship, suggesting that 
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Program Costs 

For cost information, please contact program designee. 



students practice voluntary restraint, and asking the “Cop Killer” artist what his intention was. Upon ask- 
ing, his teacher was surprised to learn that he had seen a photograph of a teenager killed by police while 
throwing grenades and wanted to warn other kids to think about the consequences of their actions. This 
led to further discussions among students and mentors about censorship, the need to inquire before 
drawing conclusions, and the power of images, presentation, and labeling, among other issues. 

Student interviews held across Vermont in 1995 by the coordinator of the Vermont Assessment Project 
revealed that students want two types of input about their work: where they stand (yardstick) and 
how to make what they are doing better (reflection and critique of work in progress). The WEB 
Project translated these ideas into a Web-based system that gathers and exhibits multimedia forms of 
student performance while simultaneously offering support for reflection and critique. Students 
report feeling more motivated because of their access to online mentors who take their work serious- 
ly. The emphasis on multiple forms of evidence of learning impels students to show what they know 
in many ways. The following quote, from a parent whose son worked on a multimedia team investi- 
gating community memories of World War n, is one illustration of the depth of engagement, complex- 
ity, and relevance of the resultant learning: 

I was thinking that if you knew Matt by whether he wrote his essays or not. . .you would think 
of him as a totally different kind of student than you get from watching him do a technology 
project. . .For me, it was a chance to see that this young man is really engaging in positive ways 
in the world which I don’t see in terms of his responses to homework and things like that. He 
came home and talked about this a lot. He related it to contemporary events. It wasn’t just 
that he learned about World War H. He talked about issues of race and gender that were 
going on at the time. He was saying that the sociocultural climate was really different in the 
thirties and forties than it is now. 

Excellence for All 

By design, the WEB Project serves all students. It contributes to excellence for all students, from tal- 
ented and gifted to students in danger of dropping out or failing. 

Organizational Change 

The WEB Project design focuses on reform at all levels. It has engaged preservice teachers and uni- 
versity faculty in online mentoring. An action research course has been developed for teachers and 
potential trainers to improve their programs through the use of technology. The challenge for organi- 
zational change even reached the evaluators, who generated a new model — based on theories of 
change, systems approaches, and participatory evaluation — of how they believe student learning is 
best mediated through technology. 



For Further Information, 

Fern Tavalin 
The WEB Project 

270 Putney Mountain Road 
Putney, Vermont 05346 



Contact: 

Telephone: 802-387-2624 

E-mail: tavalin@sover.net 

Home Page: http://www.webproject.org 
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Usefulness to Others 

The WEB Project is highly adaptable 
to any subject area and is particular- 
ly effective at overcoming traditional 
barriers of access, representation, 
socioeconomic conditions, as well as 
ethnic and gender bias. The online 
faculty room offers immediately rel- 
evant professional development in 
core content and includes mentor- 
ing in the use of technology in 
teaching and learning. The project 
also supports core discipline prac- 
tices that had been absent in the 
lives of many teachers. For exam- 
ple, prior to the WEB Project, 
Vermont music teachers neither 
composed music nor taught compo- 
sition, yet composition is the raison 
d'etre of music. Now, Vermont 
music teachers are acquiring and 
transmitting the core skills and activ- 
ities practiced by their real-world 
counterparts. 

To gain the benefits of the WEB 
Project, schools, community organi- 
zations, and state departments need 
low-cost hardware and software. A 
new project implementation with- 
out existing infrastructure would 
need to establish or secure a net- 
work connection for each partici- 
pant and mentor, establish a Web 
site with the online tools developed 
by or adapted from WEB Project 
examples, then find, train, and sup- 
port personnel to be online content 
mentors. (Small districts such as 
those found in rural states like 
Vermont should band together at a 
statewide level to increase the num- 
ber of effective professionals and 
teachers in the work; large districts 
are likely to have ample human 
resources at hand.) Four or five 
mentors may be needed in a con- 
tent area to effectively serve up to 
2,000 students and 100 teachers at 
a time, assuming that students work 
in groups and use mentors only 
when responses to work or inquiries 
about content are not available 
locally. 
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The Project has spent the last 5 years building capacity so that current participants would have the skills need- 
ed to continue in the absence of federal funding. People who use the multimedia and telecommunications 
tools well can now be found in all locations around Vermont, providing people-to-people contact when needed 
for new users. 

Perhaps the primary contribution of the WEB Project has been the creation of a permanent online program 
enhancement for all schools in Vermont, which consists of a virtual space where student and teacher work can 
be critiqued by the professionals in the arts and humanities community. 



Evidence of Effectiveness 

A variety of measures were used to evaluate the effectiveness of the WEB Project. These included an online sur- 
vey, student survey, classroom observations, document and artifact analyses, teacher focus groups and inter- 
views, student focus groups, and interviews with curriculum coordinators, initiative coordinators, and program 
coordinators. In addition, student results were measured by teacher-created student product assessments and 
selected process rubrics. 

The research associated with the WEB Project demonstrates that students and collaborating organizations have 
engaged in rich and substantive online discussion. The project has helped school districts in the state to align 
instruction with the Vermont State Frameworks. As a result of their participation in the WEB Project, students 
improve their technology skills as well as their performance in the arts by engaging in discussion with mentor 
experts. There is greater student engagement in learning tasks when students are using the technology to 
design and deliver products and performances than when working on traditional classroom projects or assign- 
ments. In addition, students exhibited increased time on task. 

More than half of the students used the technology to ask experts for feedback; 65 percent revised their prod- 
ucts at least once based on critiques and feedback from experts and mentors. Measures taken at one school 
where control and experimental groups for arts and technology were observed revealed test score gains in 
problem solving for low-performing math students. 

Student work was recognized in a host of arenas. One student’s videotape of the town’s activity to clean up the 
river was broadcast on a local television program. Another student created portfolio pieces that were used by 
the local Chamber of Commerce. A talented senior received a full 4-year university scholarship as a result of 
his multimedia skills. Two high school students obtained part-time positions as multimedia designers with 
local corporations after school and during the summer. Two teachers at different schools in one district 
observed improvements in social behavior, attendance, morale, and attention span for several troubled stu- 
dents, which transferred to other core courses. And finally, one failing student who was repeating his senior 
year obtained an internship with a local artist and has had several offers to buy his work. 
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Further Reflections by the Panel 



The following section reflects the work and thinking of the Educational Technology Expert Panel after the completion of 
the review process and the recommendation of programs to the Secretary. The material in this section should not be 
construed or interpreted as an endorsement by the U.S. Department of Education. After much discussion, the Panel 
leaves with you a definition of significant learning for your study and consideration. The Panel also offers for your con- 
sideration and use the following framework and guidance as you plan or review your educational technology program. 
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You may have noted that “learning” is cited many times throughout the Educational Technology Expert Panel’s 
“Guidelines and Materials for Submitting Educational Technology Programs for Review” (June 1999)- The Panel applied 
the criterion of complex learning and thinking skills as outlined on page 8 of this Report in making its program recom- 
mendations. After the proceedings, however, the Panel concluded that it needed to propose to the field a much more 
encompassing term than complex learning; thus, it offers to the field the term significant learning. As stated in the 
Report (page 2), “The objective of using educational technology should be to create an instructional environment that 
fosters significant learning for preK-12 students.” The Panel’s working definition of the term follows: 

Significant learning is learning that stimulates the student to further inquiry and reflection, a process 
that teaches the student how to master ever-deeper levels of understanding. It entails a combination of 
mastery of basic skills (such as number facts and literacy) and acquisition of foundational information. 
Fundamental skills and knowledge include facility with and understanding of quantitative relationships; 
the ability to recognize and construct a persuasive argument using evidence based on observation and the 
vocabulary of a specific discipline; and sufficient familiarity with the natural and historical world to rec- 
ognize patterns of behavior, causality, and meaning. Significant learning enables the student to know 
what questions to ask when confronting new information or circumstances and how to marshal appro- 
priate processes and resources to solve problems. 

Significant learning is ultimately what the teaching and learning process is aiming to accomplish with all learners in our 
nation’s schools. 
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REVISITING THE CRITERIA FOR EVALUATING 
TECHNOLOGY PROGRAMS 
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The preceding criteria and indicators (pages 7-10) were used to evaluate the educational technology programs submit- 
ted to the Expert Panel. Based on the knowledge gained by the Panel and as a result of a careful examination of the 
review process, the Panel proposes the following refinements in the criteria and indicators for greater clarity. You will 
also see below a set of rubrics developed and offered by the Panel to (1) clarify the scoring system; and (2) help you 
evaluate the soundness and effectiveness of your educational technology program. The Panel hopes that this refinement 
in the indicators and its proposed set of rubrics will help to stimulate meaningful conversations and thinking around 
what constitutes effective teaching and student learning in our nation’s schools when technology is used as part of the 
educational process. 

A Quality of Program 

Criterion 1. The program addresses an important educational issue or issues and 
articulates its goals and design clearly. 

Please describe the program in detail. Readers should be able to understand the overall program, as well as 
what participants actually do, sufficiently well to explain the program to others. Include the items following each 
bullet point. 

O The educational goals are significant. Include in description the 

• need or problem the program addresses and how it relates to teaching and learning 
in preK-12 schools; 

• program goals; and 

• content and learning goals. 

□ The program design is thoughtful and coherent. Include in description the 

• program design — how the parts relate to the whole; 

• technology used and how it is essential for achieving the program’s goals; 

• professional development provided as part of the program; and 

• assessments used to determine the program’s efficacy and achievements. 

□ The program goals and design are supported by research. Include in description the 

• relationship of goals and design to relevant research on educational reform, the science 
of teaching and learning, or technology. 

□ The program description is clear and complete. It should include 

• key learning activities for participants; 

• subject population (s), including ethnic, racial, socioeconomic, and gender percentages and 
the size of any special populations served (e.g., ESL, AP biology students, and students with 
disabilities); 

• overall size and maintenance costs (e.g., funding and staff requirements, and 
number of people in target population); 

• keys to the program’s success; and 

• a specific, concrete example that best captures the changes achieved by this program. 
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Criterion 1 Rubrics 



Level 1 There is little or no demonstration that the program addresses important educational issues or articu- 
lates its goals and design clearly. The program’s goals, design, and description are vague and incomplete. The 
design for the use of technology may be an example of already-existing widespread practice and therefore will 
not significantly advance or inform the field. The design may be vague, incoherent, unclear, or unsupported by 
research; the program description may be unclear. In most cases no indicator is adequately addressed. 



Level 2 There is insufficient or incomplete demonstration that the program addresses important educational 
issues or articulates its goals and design clearly The goals, program design, and program description are often 
too general to be considered adequate. In some cases, however, the program is designed or described ade- 
quately, but the goals are not significant. The goals may be limited or local — not set in a broader educational or 
research context. Generally, only one indicator is adequately addressed. 



Level 3 The goals are adequately significant and the design adequately thoughtful and coherent. The descrip- 
tion, however, may be insufficiently clear or incomplete. The goals and design are often not supported by 
research. At least two indicators, including the significance of the educational goals, are adequately addressed. 



Level 4 The goals, design, and description may be convincingly significant, thoughtful, coherent, clear, and 
complete. Alternatively, the goals and design may be compelling in their significance and thoughtfulness, but the 
description only adequate in its clarity and completeness. Program goals and designs are at least adequately 
supported by research. 



Level 5 The goals, design, and description are convincing or compelling. The program goals and designs are 
convincingly supported by research. 



B. Educational Significance 



The Expert Panel considers the following three areas — learning, equity, and organizational change — 
essential to fulfilling the promise of educational technology. A sound program must address all three, and 
all three must be shown to have impact on or linkage to preK-12 student learning. 

Criterion 2. The program develops complex learning and thinking skills. 

If the participants are other than preK-12 students, the applicant should articulate the program's goals 
and their connection to student learning. 

O Students increase their in-depth understanding and competence in at least one content discipline. 

□ Students develop the habits of self-directed, lifelong learning (e.g., the ability to collaborate, to 
direct one’s own learning, to solve problems, to communicate ideas clearly, and to think flexibly and 
critically). 

□ Students become proficient and critical consumers and producers of educational technology. 

□ Students are prepared to enter a technology-infused workplace. 
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Criterion 2 Rubrics 



Level 1 There is little or no demonstration that the program develops complex learning and thinking skills. 

It may be unclear just what activities learners engage in, and how and what they may be learning from these 
activities. The contribution of technology to the learning may not be well specified. At this level, there is insuffi- 
cient demonstration of any of the indicators. 



Level 2 There is insufficient demonstration that the program develops complex learning and thinking skills. 

The program may adequately demonstrate that it is addressing one indicator, or may provide more vague 
demonstration of more than one indicator. Programs may describe clearly the activities learners engage in, with- 
out making the case for how these activities lead to the development of complex learning and thinking skills. If 
the program is for teachers, it may be unclear how the program can lead to their students’ developing complex 
learning and thinking skills. Programs that focus on the transition from school to work may not clearly demon- 
strate how they connect to content learning or to self-directed, lifelong learning. 



Level 3 There is adequate demonstration that the program develops complex learning and thinking skills. The 
program adequately addresses at least two indicators. These programs describe program activities sufficiently, 
and make a case for how these activities contribute to learning, for two of the indicators. Programs that serve 
teachers may not fully clarify how the program will result in complex learning for students. Programs that focus 
on the transition from school to work may fail to demonstrate how they connect to content learning or to self- 
directed, lifelong learning. 



Level 4 There is convincing demonstration that the program develops complex learning and thinking skills 
The program addresses a minimum of three indicators, at least one convincingly (the other two at least ade- 
quately). The program provides a clear description of program activities, and makes a case, through both argu- 
ment and examples, for how these activities contribute to learning. Programs that serve teachers clarify the links 
to student learning — that is, how teacher participation in the program is intended to result in complex learning 
for students. Programs that focus on the transition from school to work demonstrate their intended connection 
to content learning or to self-directed, lifelong learning. 



Level 5 There is compelling demonstration that the program develops complex learning and thinking skills 
The program addresses at least three indicators, at least two convincingly (the third at least adequately). The 
program provides a clear description of program activities and makes a case, through both argument and exam- 
ples, for how these activities contribute to learning. Programs that serve teachers demonstrate the links to stu- 
dent learning. Specifically, they show, through examples of student activities and student work, how teacher par- 
ticipation in the program leads to complex learning for students. Programs that focus on the transition from 
school to work demonstrate their actual connection to content learning or to self-directed, lifelong learning. 
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Criterion 3 • The program contributes to educational excellence for all . 



□ The program conveys high expectations for all learners. 

□ The program responds to the diverse needs of varied populations of learners. Learners may vary, for 
example, by achievement level, ethnicity, socioeconomic level, English language proficiency, gender, 
learning style, and handicapping conditions. 

□ The program includes active outreach and partnerships with the community or relevant organizations to 
encourage broad participation of diverse groups of learners. 

□ The program increases the participation or achievement of underserved learners so that the gaps between 
these learners and other categories of students diminishes. 
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Criterion 3 Rubrics 



Level 1 There is little or no demonstration that the program contributes to educational excellence for all. It 
may not be clear who has access to or is served by the program. There is no evidence of outreach or collabora- 
tion. While the program may, for example, set high expectations for its learners, it may serve only students who 
are already well served without attempting to include and serve a broader range of learners. 



Level 2 There is limited and insufficient demonstration that the program contributes to educational excellence 
for all. A program may address the needs of an underserved group of learners, but without conveying expecta- 
tions that are high or clear. A program in teacher development may serve a diverse group of teachers without 
making any connection to the diverse needs of the students they teach. A software-based program may claim that 
the software is designed to serve diverse learners but not demonstrate how or that it does. There is no evidence 
of active outreach or collaborative partnership. 



Level 3 There is adequate demonstration that the program contributes to educational excellence for all. 
Programs generally set high expectations and serve diverse groups of learners. Many do not have strong out- 
reach programs. Programs may have partners, but be unable to show how these partnerships contribute to 
broader participation. Some programs are able to demonstrate that they are closing gaps in the participation of 
underserved learners. 



Level 4 There is clear and convincing demonstration that the program contributes to educational excellence for 
all. Programs set high expectations for all learners, meet the needs of diverse and underserved learners, and 
have active outreach and collaborative partnerships. Programs show that they narrow the gaps in participation 
for underserved learners, although at this level they do not yet demonstrate an ability to narrow the gaps in stu- 
dent performance. 



Level 5 There is complete and compelling demonstration that the program contributes to educational excel- 
lence for all. The program fully addresses all four indicators. Programs are able to demonstrate that they have 
increased both the participation and the performance of underserved groups of learners. 
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Criterion 4. The program promotes coherent organizational chang e. 



□ The program reflects a vision of educational renewal consistent with disciplinary content standards, 
recommendations from national commissions, findings from educational research, and documented best 
practices. 

□ The program has identified a set of goals that will lead to educational renewal within the organization. 

□ The program has identified and involved key constituents related to its set of goals — parents, professional 
groups, and community members. 

□ Through partnerships and professional development, the program enhances the human capacity necessary 
to accomplish its goals (e.g., allocates time for teachers’ and administrators’ collaboration and planning). 

□ As a result of the program, policies, procedures, funding, and practice have been changed to increase their 
alignment in support of sustainable change. 



Criterion 4 Rubrics 



Level 1 There is little or no demonstration that the program promotes coherent organizational change. Either 
the response to this criterion is too vague, or the program has not demonstrated the vision, goals, involvement of 
constituencies, enhancement of human capacity, or changes in policy that can promote organizational change. 



Level 2 There is limited and insufficient demonstration that the program promotes coherent organizational 
change. For example, a program may claim to be guided by a vision without making clear what that vision is, or 
may recognize the need for capacity building to support change without actually building that capacity. 
Alternately, it may indicate that it has established partnerships without showing how they contribute to organiza- 
tional change. The program may adequately demonstrate one indicator. 



Level 3 There is adequate demonstration that the program promotes some organizational change. The pro- 
gram adequately addresses at least three of the indicators in ways that can result in organizational change. 
Change is not yet comprehensive or fully coherent. 



Level 4 There is clear and convincing demonstration that the program promotes coherent organizational 
change. The program addresses four indicators, at least three convincingly. 



Level 5 There is complete and compelling demonstration that the program promotes coherent organizational 
change. All five indicators are addressed, at least four convincingly. 
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C. Evidence of Effectiveness 



Criterion 5. The program has rigorous, measurable evidence of its achievements for one or more 
among Criteria 2, 3, and 4 (learning, equity, and organizational change). 

□ The evidence is clearly related to the claims made for program effectiveness. For example: 

• framework for gathering evidence is closely tied to program goals; and 

• evidence bears directly on claims made for program effectiveness. 

□ The design for collecting evidence meets generally accepted standards of quality and may include: 

• one or more comparison groups; 

• a quantified validation by an external authority; 

• a formal evaluation; 

• a quantified demonstration of positive change among participants as a result of the program 

(e.g., increased parental involvement in school governance; diminished gaps in achievement between 
groups; increased enrollment in rigorous mathematics courses or graduation rates among subject 
populations; changes in the base funding and requirements for professional development); and 

• an in-depth, qualitative analysis of change among participants as a result of the program 
(e.g., case studies, ethnographies, principled analysis of observations and interviews). 

□ The evidence is complete and convincing. This includes the following: 

• methods are fully described and explained; 

• sample sizes are appropriate; 

• results are clearly and completely documented; 

• data are disaggregated for relevant groups; 

• multiple methods are used to support key claims; and 

• results are generally positive. 




Criterion 5 Rubrics 



Level 1 There is no rigorous, measurable evidence for the program’s achievements. If evidence is presented, 
it is not clearly related to program goals and claims, does not meet standards of quality, and is often anecdotal. 
Programs at this level are often in too early a stage to have collected valid evidence, although they may have 
plans for doing so. 



Level 2 There is incomplete, insufficient evidence for the program’s achievements. Although considerable 
amounts of data may be presented, the data do not constitute credible evidence of effectiveness. Often the pro- 
gram lacks a framework for gathering, analyzing, and interpreting evidence. A program may make numerous 
claims, for example, but present data that bear on only one or two, or none, of them. A program may be very 
limited in the kind of data it collects; it may, for example, rely solely on attitudinal evidence. A program may not 
sample adequately given the size and type of groups it serves. A program may not describe its methods with suf- 
ficient clarity. Generally, programs use only one method to support any claim. Often, programs are in too early 
a stage of development to present credible evidence of impact. 



Level 3 There is credible, adequate evidence for the program’s achievements. The evidence is clearly related 
to program goals and to claims of effectiveness. The evaluation design meets adequate standards of quality. 

The design includes most of the elements required to make valid inferences about the effectiveness of the pro- 
gram. The evidence is sufficiently well documented, analyzed, and complete to present an adequate case for the 
program’s effectiveness with respect to at least one criterion. 



Level 4 There is convincing evidence for the program’s achievements. The evidence is clearly related to pro- 
gram goals and to claims for effectiveness. The evaluation design meets high standards of quality. The design 
includes most of the elements required to make valid inferences about the effectiveness of the program. The 
evidence is well documented, carefully analyzed, and complete. It presents a convincing case for the program’s 
effectiveness with respect to at least one criterion. It further presents an adequate case for the effectiveness of 
one other criteria. 



Level 5 There is compelling evidence for the program’s achievements. The evidence is clearly related to pro- 
gram goals and to claims for effectiveness. The evaluation design is driven by these goals and claims. For all 
claims, clear and appropriate evidence is provided. The design meets high standards of quality. The design 
includes all the elements required to make valid inferences about the effectiveness of the program. The evidence 
is completely documented, carefully analyzed, and presents a compelling case for the program’s effectiveness 
with respect to at least one criterion. A program at this level shows evidence of at least adequate effectiveness for 
the other two criteria. 
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D. Usefulness to Others 



Criterion 6. The program is adaptable for use in multiple contexts. 

□ The program’s technology requirements are easily available to potential users. 

□ The program is cost-effective relative to its benefits. 

□ After its initial implementation, the program is sustainable with existing resources (i.e., does not require 
extraordinary/unreasonable time, effort, or binding), and scalable (i.e., can naturally expand its scope to 
several teachers, multiple grade levels/subjects/sites, and different disciplines). 

□ The program is adaptable to a range of educational settings. 

□ The program provides clear and detailed guidelines about the conditions required for its successful 
implementation (e.g., philosophical assumptions and level of personnel training prior to start-up). 

□ The program provides clear and detailed guidelines about the start-up and annual ongoing costs required 
for its successful implementation (e.g., equipment and infrastructure; personnel; training costs; technical 
support; replacement or upgraded hardware or software; materials and supplies). 



t. 
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Criterion 6 Rubrics 



Level 1 There is little or no demonstration that the program is adaptable for use in multiple contexts. In some 
cases, the program is too new to be able to show adaptability. In some cases, the program description is so vague 
or unfocused that what would be replicated or sustained is not clear. 



Level 2 There is limited or insufficient demonstration that the program is adaptable for use in multiple contexts. 
Programs may demonstrate that their technology and cost requirements are reasonable and that they are locally sus- 
tainable, but not that they are scalable or adaptable to a range of settings; or they may provide no guidelines for 
implementation. Programs at this level may have made no effort to expand beyond a set of limited goals for a par- 
ticular target population. Alternatively, they may be prohibitively costly, although their model is, in principle, adapt- 
able. 



Level 3 There is adequate demonstration that the program is adaptable for use in multiple contexts. Programs at 
this level have adequately available technology, are cost effective, and demonstrate adequate scalability or adaptabili- 
ty. Some may have clear guidelines for implementation, including costs and other requirements. 



Level 4 There is convincing demonstration that the program is adaptable for use in multiple contexts. Programs 
at this level are at least adequate with respect to five of the she indicators. 



Level 5 There is compelling demonstration that the program is adaptable for use in multiple contexts. Programs 
address all of the indicators and are able to show that they can be been widely used in multiple settings. 







so 



The purpose of the Expert Panel is to help practitioners make better-informed decisions by setting forth the 
experience and insights of others. In the last decade, technology has achieved a critical mass of availability 
and power. We stand at the opening of a revolutionary pathway built on interactive capabilities. No one 
knows where this pathway will ultimately lead, but many have contributed to the development of a reliable 
compass to guide us all forward. 
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